Struggling to get started with machine learning using Python? In this step-by-step, hands-on tutorial you will learn how to perform machine learning using Python on numerical data and image data.

By the time you are finished reading this post, you will be able to get your start in machine learning.

To launch your machine learning in Python education, just keep reading!

Inside this tutorial, you will learn how to perform machine learning in Python on numerical data and image data.

You will learn how to operate popular Python machine learning and deep learning libraries, including two of my favorites:

Specifically, you will learn how to:

Using this technique you will be able to get your start with machine learning and Python!

Along the way, you’ll discover popular machine learning algorithms that you can use in your own projects as well, including:

This hands-on experience will give you the knowledge (and confidence) you need to apply machine learning in Python to your own projects.

Before we can get started with this tutorial you first need to make sure your system is configured for machine learning. Today’s code requires the following libraries:

Each of these can be installed in your environment (virtual environments recommended) with pip:

In order to help you gain experience performing machine learning in Python, we’ll be working with two separate datasets.

The first one, the Iris dataset, is the machine learning practitioner’s equivalent of “Hello, World!” (likely one of the first pieces of software you wrote when learning how to program).

The second dataset, 3-scenes, is an example image dataset I put together — this dataset will help you gain experience working with image data, and most importantly, learn what techniques work best for numerical/categorical datasets vs. image datasets.

Let’s go ahead and get a more intimate look at these datasets.

The Iris dataset is arguably one of the most simplistic machine learning datasets — it is often used to help teach programmers and engineers the fundamentals of machine learning and pattern recognition.

We call this dataset the “Iris dataset” because it captures attributes of three Iris flower species:

Each species of flower is quantified via four numerical attributes, all measured in centimeters:

Our goal is to train a machine learning model to correctly predict the flower species from the measured attributes.

It’s important to note that one of the classes is linearly separable from the other two — the latter are not linearly separable from each other.

In order to correctly classify these the flower species, we will need a non-linear model.

It’s extremely common to need a non-linear model when performing machine learning with Python in the real world — the rest of this tutorial will help you gain this experience and be more prepared to conduct machine learning on your own datasets.

The second dataset we’ll be using to train machine learning models is called the 3-scenes dataset and includes 948 total images of 3 scenes:

The 3-scenes dataset was created by sampling the 8-scenes dataset from Oliva and Torralba’s 2001 paper, Modeling the shape of the scene: a holistic representation of the spatial envelope.

Our goal will be to train machine learning and deep learning models with Python to correctly recognize each of these scenes.

I have included the 3-scenes dataset in the “Downloads” section of this tutorial. Make sure you download the dataset + code to this blog post before continuing.

Whenever you perform machine learning in Python I recommend starting with a simple 5-step process:

This pipeline will evolve as your machine learning experience grows, but for beginners, this is the machine learning process I recommend for getting started.

To start, we must examine the problem.

Ask yourself:

The last question, in particular, is critical — the more you apply machine learning in Python, the more experience you will gain.

Based on your previous experience you may already know an algorithm that works well.

From there, you need to prepare your data.

Typically this step involves loading your data from disk, examining it, and deciding if you need to perform feature extraction or feature engineering.

Feature extraction is the process of applying an algorithm to quantify your data in some manner.

For example, when working with images we may wish to compute histograms to summarize the distribution of pixel intensities in the image — in this manner, we can characterize the color of the image.

Feature engineering, on the other hand, is the process of transforming your raw input data into a representation that better represents the underlying problem.

Feature engineering is a more advanced technique and one I recommend you explore once you already have some experience with machine learning and Python.

Next, you’ll want to spot-check a set of algorithms.

What do I mean by spot-checking?

Simply take a set of machine learning algorithms and apply them to the dataset!

You’ll likely want to stuff the following machine learning algorithms in your toolbox:

Try to bring a robust set of machine learning models to the problem — your goal here is to gain experience on your problem/project by identifying which machine learning algorithms performed well on the problem and which ones did not.

Once you’ve defined your set of models, train them and evaluate the results.

Which machine learning models worked well? Which models performed poorly?

Take your results and use them to double-down your efforts on the machine learning models that performed while discarding the ones that didn’t.

Over time you will start to see patterns emerge across multiple experiments and projects.

You’ll start to develop a “sixth sense” of what machine learning algorithms perform well and in what situation.

For example, you may discover that Random Forests work very well when applied to projects that have many real-valued features.

On the other hand, you might note that Logistic Regression can handle sparse, high-dimensional spaces well.

You may even find that Convolutional Neural Networks work great for image classification (which they do).

Use your knowledge here to supplement traditional machine learning education — the best way to learn machine learning with Python is to simply roll up your sleeves and get your hands dirty!

A machine learning education based on practical experience (supplemented with some super basic theory) will take you a long way on your machine learning journey!

Now that we have discussed the fundamentals of machine learning, including the steps required to perform machine learning in Python, let’s get our hands dirty.

In the next section, we’ll briefly review our directory and project structure for this tutorial.

Note: I recommend you use the “Downloads” section of the tutorial to download the source code and example data so you can easily follow along.

Once we’ve reviewed the directory structure for the machine learning project we will implement two Python scripts:

As a bonus we’ll implement two more Python scripts, each of these dedicated to neural networks and deep learning:

Let’s get started by first reviewing our project structure.

Be sure to grab the “Downloads” associated with this blog post.

From there you can unzip the archive and inspect the contents:

The Iris dataset is built into scikit-learn. The 3-scenes dataset, however, is not. I’ve included it in the 3scenes/ directory and as you can see there are three subdirectories (classes) of images.

We’ll be reviewing four Python machine learning scripts today:

The first script we are going to implement is classify_iris.py — this script will be used to spot-check machine learning algorithms on the Iris dataset.

Once implemented, we’ll be able to use classify_iris.py to run a suite of machine learning algorithms on the Iris dataset, look at the results, and decide on which algorithm works best for the project.

Let’s get started — open up the classify_iris.py file and insert the following code:

Lines 2-12 import our required packages, specifically:

Using argparse , let’s parse a single command line argument flag, --model on Lines 15-18. The --model switch allows us to choose from any of the following models:

The models dictionary on Lines 23-31 defines the suite of models we will be spot-checking (we’ll review the results of each of these algorithms later in the post):

The keys can be entered directly in the terminal following the --model switch. Here’s an example:

From there the KNeighborClassifier will be loaded automatically. This conveniently allows us to call any one of 7 machine learning models one-at-a-time and on demand in a single Python script (no editing the code required)!

Moving on, let’s load and split our data:

Our dataset is easily loaded with the dedicated load_iris method on Line 36. Once the data is in memory, we go ahead and call train_test_split to separate the data into 75% for training and 25% for testing (Lines 37 and 38).

The final step is to train and evaluate our model:

Lines 42 and 43 train the Python machine learning model (also known as “fitting a model”, hence the call to .fit ).

From there, we evaluate the model on the testing set (Line 47) and then print a classification_report to our terminal (Lines 48 and 49).

The following script, classify_images.py , is used to train the same suite of machine learning algorithms above, only on the 3-scenes image dataset.

It is very similar to our previous Iris dataset classification script, so be sure to compare the two as you follow along.

Let’s implement this script now:

First, we import our necessary packages on Lines 2-16. It looks like a lot, but you’ll recognize most of them from the previous script. The additional imports for this script include:

You’ll see how each of the imports is used in the coming lines of code.

Next let’s define a function called extract_color_stats :

Most machine learning algorithms perform very poorly on raw pixel data. Instead, we perform feature extraction to characterize the contents of the images.

Here we seek to quantify the color of the image by extracting the mean and standard deviation for each color channel in the image.

Given three channels of the image (Red, Green, and Blue), along with two features for each (mean and standard deviation), we have 3 x 2 = 6 total features to quantify the image. We form a feature vector by concatenating the values.

In fact, that’s exactly what the extract_color_stats function is doing:

We’ll be using this function to calculate a feature vector for each image in the dataset.

Let’s go ahead and parse two command line arguments:

Where the previous script had one argument, this script has two command line arguments:

Again, we have seven machine learning models to choose from with the --model argument:

After defining the models dictionary, we’ll need to go ahead and load our images into memory:

Our imagePaths are extracted on Line 53. This is just a list of the paths themselves, we’ll load each actual image shortly.

I’ve defined two lists, data and labels (Lines 54 and 55). The data list will hold our image feature vectors and the class labels corresponding to them. Knowing the label for each image allows us to train our machine learning model to automatically predict class labels for our test images.

Lines 58-68 consist of a loop over the imagePaths in order to:

Now, let’s encode our labels and construct our data splits:

Our textual labels are transformed into an integer representing the label using the LabelEncoder (Lines 71 and 72):

Just as in our Iris classification script, we split our data into 75% for training and 25% for testing (Lines 76 and 77).

Finally, we can train and evaluate our model:

These lines are nearly identical to the Iris classification script. We’re fitting (training) our model and evaluating it (Lines 81-86). A classification_report is printed in the terminal so that we can analyze the results (Lines 87 and 88).

Speaking of results, now that we’re finished implementing both classify_irs.py and classify_images.py , let’s put them to the test using each of our 7 Python machine learning algorithms.

The k-Nearest Neighbors classifier is by far the most simple image classification algorithm.

In fact, it’s so simple that it doesn’t actually “learn” anything. Instead, this algorithm relies on the distance between feature vectors. Simply put, the k-NN algorithm classifies unknown data points by finding the most common class among the k closest examples.

Each data point in the k closest data points casts a vote and the category with the highest number of votes wins!

Or, in plain English: “Tell me who your neighbors are, and I’ll tell you who you are.”

For example, in Figure 6 above we see three sets of our flowers:

We have plotted each of the flower images according to their lightness of the petals (color) and the size of the petals (this is an arbitrary example so excuse the non-formality).

We can clearly see that the image is a sunflower, but what does k-NN think given our new image is equal distance to one pansy and two sunflowers?

Well, k-NN would examine the three closest neighbors (k=3) and since there are two votes for sunflowers versus one vote for pansies, the sunflower class would be selected.

To put k-NN in action, make sure you’ve used the “Downloads” section of the tutorial to download the source code and example datasets.

From there, open up a terminal and execute the following command:

Here you can see that k-NN is obtaining 95% accuracy on the Iris dataset, not a bad start!

Let’s look at our 3-scenes dataset:

On the 3-scenes dataset, the k-NN algorithm is obtaining 75% accuracy.

In particular, k-NN is struggling to recognize the “highway” class (~56% accuracy).

We’ll be exploring methods to improve our image classification accuracy in the rest of this tutorial.

For more information on how the k-Nearest Neighbors algorithm works, be sure to refer to this post.

After k-NN, Naïve Bayes is often the first true machine learning algorithm a practitioner will study.

The algorithm itself has been around since the 1950s and is often used to obtain baselines for future experiments (especially in domains related to text retrieval).

The Naïve Bayes algorithm is made possible due to Bayes’ theorem (Figure 7).

Essentially, Naïve Bayes formulates classification as an expected probability.

Given our input data, D, we seek to compute the probability of a given class, C.

Formally, this becomes P(C | D).

To actually compute the probability we compute the numerator of Figure 7 (ignoring the denominator).

The expression can be interpreted as:

Let’s go ahead and apply the Naïve Bayes algorithm to the Iris dataset:

We are now up to 98% accuracy, a marked increase from the k-NN algorithm!

Now let’s apply Naïve Bayes to the 3-scenes dataset for image classification:

Uh oh!

It looks like we only obtained 66% accuracy here.

Does that mean that k-NN is better than Naïve Bayes and that we should always use k-NN for image classification?

Not so fast.

All we can say here is that for this particular project and for this particular set of extracted features the k-NN machine learning algorithm outperformed Naive Bayes.

We cannot say that k-NN is better than Naïve Bayes and that we should always use k-NN instead.

Thinking that one machine learning algorithm is always better than the other is a trap I see many new machine learning practitioners fall into — don’t make that mistake.

For more information on the Naïve Bayes machine learning algorithm, be sure to refer to this excellent article.

Figure 8: Logistic Regression is a machine learning algorithm based on a logistic function always in the range [0, 1]. Similar to linear regression, but based on a different function, every machine learning and Python enthusiast needs to know Logistic Regression (source).The next machine learning algorithm we are going to explore is Logistic Regression.

Logistic Regression is a supervised classification algorithm often used to predict the probability of a class label (the output of a Logistic Regression algorithm is always in the range [0, 1]).

Logistic Regression is heavily used in machine learning and is an algorithm any machine learning practitioner needs Logistic Regression in their Python toolbox.

Let’s apply Logistic Regression to the Iris dataset:

Here we are able to obtain 98% classification accuracy!

And furthermore, note that both the Setosa and Versicolor classes are classified 100% correctly!

Now let’s apply Logistic Regression to the task of image classification:

Logistic Regression performs slightly better than Naive Bayes here, obtaining 69% accuracy but in order to beat k-NN we’ll need a more powerful Python machine learning algorithm.

Support Vector Machines (SVMs) are extremely powerful machine learning algorithms capable of learning separating hyperplanes on non-linear datasets through the kernel trick.

If a set of data points are not linearly separable in an N-dimensional space we can project them to a higher dimension — and perhaps in this higher dimensional space the data points are linearly separable.

The problem with SVMs is that it can be a pain to tune the knobs on an SVM to get it to work properly, especially for a new Python machine learning practitioner.

When using SVMs it often takes many experiments with your dataset to determine:

If, at first, your SVM is not obtaining reasonable accuracy you’ll want to go back and tune the kernel and associated parameters — tuning those knobs of the SVM is critical to obtaining a good machine learning model. With that said, let’s apply an SVM to our Iris dataset:

Just like Logistic Regression, our SVM obtains 98% accuracy — in order to obtain 100% accuracy on the Iris dataset with an SVM, we would need to further tune the parameters to the kernel.

Let’s apply our SVM to the 3-scenes dataset:

Wow, 83% accuracy!

That’s the best accuracy we’ve seen thus far!

Clearly, when tuned properly, SVMs lend themselves well to non-linearly separable datasets.

The basic idea behind a decision tree is to break classification down into a set of choices about each entry in our feature vector.

We start at the root of the tree and then progress down to the leaves where the actual classification is made.

Unlike many machine learning algorithms such which may appear as a “black box” learning algorithm (where the route to the decision can be hard to interpret and understand), decision trees can be quite intuitive — we can actually visualize and interpret the choice the tree is making and then follow the appropriate path to classification.

For example, let’s pretend we are going to the beach for our vacation. We wake up the first morning of our vacation and check the weather report — sunny and 90 degrees Fahrenheit.

That leaves us with a decision to make: “What should we do today? Go to the beach? Or see a movie?”

Subconsciously, we may solve the problem by constructing a decision tree of our own (Figure 10).

First, we need to know if it’s sunny outside.

A quick check of the weather app on our smartphone confirms that it is indeed sunny.

We then follow the Sunny=Yes branch and arrive at the next decision — is it warmer than 70 degrees out?

Again, after checking the weather app we can confirm that it will be > 70 degrees outside today.

Following the >70=Yes branch leads us to a leaf of the tree and the final decision — it looks like we are going to the beach!

Internally, decision trees examine our input data and look for the best possible nodes/values to split on using algorithms such as CART or ID3. The tree is then automatically built for us and we are able to make predictions.

Let’s go ahead and apply the decision tree algorithm to the Iris dataset:

Our decision tree is able to obtain 95% accuracy.

What about our image classification project?

Here we obtain 74% accuracy — not the best but certainly not the worst either.

Since a forest is a collection of trees, a Random Forest is a collection of decision trees.

However, as the name suggestions, Random Forests inject a level of “randomness” that is not present in decision trees — this randomness is applied at two points in the algorithm.

In practice, injecting randomness into the Random Forest classifier by bootstrapping training samples for each tree, followed by only allowing a subset of the features to be used for each tree, typically leads to a more accurate classifier.

At prediction time, each decision tree is queried and then the meta-Random Forest algorithm tabulates the final results.

Let’s try our Random Forest on the Iris dataset:

As we can see, our Random Forest obtains 96% accuracy, slightly better than using just a single decision tree.

But what about for image classification?

Do Random Forests work well for our 3-scenes dataset?

Using a Random Forest we’re able to obtain 84% accuracy, a full 10% better than using just a decision tree.

In general, if you find that decision trees work well for your machine learning and Python project, you may want to try Random Forests as well!

One of the most common neural network models is the Perceptron, a linear model used for classification.

A Perceptron accepts a set of inputs, takes the dot product between the inputs and the weights, computes a weighted sum, and then applies a step function to determine the output class label.

We typically don’t use the original formulation of Perceptrons as we now have more advanced machine learning and deep learning models. Furthermore, since the advent of the backpropagation algorithm, we can train multi-layer Perceptrons (MLP).

Combined with non-linear activation functions, MLPs can solve non-linearly separable datasets as well.

Let’s apply a Multi-layer Perceptron machine learning algorithm to our Iris dataset using Python and scikit-learn:

Our MLP performs well here, obtaining 98% classification accuracy.

Let’s move on to image classification with an MLP:

The MLP reaches 81% accuracy here — quite respectable given the simplicity of the model!

If you’re interested in machine learning and Python then you’ve likely encountered the term deep learning as well.

What exactly is deep learning?

And what makes it different than standard machine learning?

Well, to start, it’s first important to understand that deep learning is a subfield of machine learning, which is, in turn, a subfield of the larger Artificial Intelligence (AI) field.

The term “deep learning” comes from training neural networks with many hidden layers.

In fact, in the 1990s it was extremely challenging to train neural networks with more than two hidden layers due to (paraphrasing Geoff Hinton):

It’s a different story now. We now have:

All of this has culminated at exactly the right time to give rise to the latest incarnation of deep learning.

And chances are, if you’re reading this tutorial on machine learning then you’re most likely interested in deep learning as well!

To gain some experience with neural networks, let’s implement one using Python and Keras.

Open up the nn_iris.py and insert the following code:

Let’s import our packages.

Our Keras imports are for creating and training our simple neural network (Lines 2-4). You should recognize the scikit-learn imports by this point (Lines 5-8).

We’ll go ahead and load + split our data and one-hot encode our labels on Lines 13-20. A one-hot encoded vector consists of binary elements where one of them is “hot” such as [0, 0, 1] or [1, 0, 0] in the case of our three flower classes.

Now let’s build our neural network:

Our neural network consists of two fully connected layers using sigmoid activation.

The final layer has a “softmax classifier” which essentially means that it has an output for each of our classes and the outputs are probability percentages.

Let’s go ahead and train and evaluate our model :

Our model is compiled on Lines 30-32 and then the training is initiated on Lines 33 and 34.

Just as with our previous two scripts, we’ll want to check on the performance by evaluating our network. This is accomplished by making predictions on our testing data and then printing a classification report (Lines 38-40).

There’s a lot going on under the hood in these short 40 lines of code. For an in-depth walkthrough of neural network fundamentals, please refer to the Starter Bundle of Deep Learning for Computer Vision with Python or the PyImageSearch Gurus course.

We’re down to the moment of truth — how will our neural network perform on the Iris dataset?

Wow, perfect! We hit 100% accuracy!

This neural network is the first Python machine learning algorithm we’ve applied that’s been able to hit 100% accuracy on the Iris dataset.

The reason our neural network performed well here is because we leveraged:

Given that our neural network performed so well on the Iris dataset we should assume similar accuracy on the image dataset as well, right? Well, we actually have a trick up our sleeve — to obtain even higher accuracy on image datasets we can use a special type of neural network called a Convolutional Neural Network.

Convolutional Neural Networks, or CNNs for short, are special types of neural networks that lend themselves well to image understanding tasks. Unlike most machine learning algorithms, CNNs operate directly on the pixel intensities of our input image — no need to perform feature extraction!

Internally, each convolution layer in a CNN is learning a set of filters. These filters are convolved with our input images and patterns are automatically learned. We can also stack these convolution operates just like any other layer in a neural network.

Let’s go ahead and learn how to implement a simple CNN and apply it to basic image classification.

Open up the basic_cnn.py script and insert the following code:

In order to build a Convolutional Neural Network for machine learning with Python and Keras, we’ll need five additional Keras imports on Lines 2-8.

This time, we’re importing convolutional layer types, max pooling operations, different activation functions, and the ability to flatten. Additionally, we’re using the Adam optimizer rather than SGD as we did in the previous simple neural network script.

You should be acquainted with the names of the scikit-learn and other imports by this point.

This script has a single command line argument, --dataset . It represents the path to the 3-scenes directory on disk again.

Let’s load the data now:

Similar to our classify_images.py script, we’ll go ahead and grab our imagePaths and build our data and labels lists.

There’s one caveat this time which you should not overlook:

We’re operating on the raw pixels themselves rather than a color statistics feature vector. Take the time to review classify_images.py once more and compare it to the lines of basic_cnn.py .

In order to operate on the raw pixel intensities, we go ahead and resize each image to 32×32 and scale to the range [0, 1] by dividing by 255.0 (the max value of a pixel) on Lines 36 and 37. Then we add the resized and scaled image to the data list (Line 38).

Let’s one-hot encode our labels and split our training/testing data:

And then build our image classification CNN with Keras:

On Lines 55-67, demonstrate an elementary CNN architecture. The specifics aren’t important right now, but if you’re curious, you should:

Let’s go ahead and train + evaluate our CNN model:

Our model is trained and evaluated similarly to our previous script.

Let’s give our CNN a try, shall we?

Using machine learning and our CNN we are able to obtain 92% accuracy, far better than any of the previous machine learning algorithms we’ve tried in this tutorial!

Clearly, CNNs lend themselves very well to image understanding problems.

On the surface, you may be tempted to look at the results of this post and draw conclusions such as:

Be careful with those types of conclusions and keep in mind the 5-step machine learning process I detailed earlier in this post:

Each and every problem you encounter is going to be different in some manner.

Over time, and through lots of hands-on practice and experience, you will gain a “sixth sense” as to what machine learning algorithms will work well in a given situation.

However, until you reach that point you need to start by applying various machine learning algorithms, examining what works, and re-doubling your efforts on the algorithms that showed potential.

No two problems will be the same and, in some situations, a machine learning algorithm you once thought was “poor” will actually end up performing quite well!

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

Click here to join PyImageSearch University

In this tutorial, you learned how to get started with machine learning and Python.

Specifically, you learned how to train a total of nine different machine learning algorithms:

We then applied our set of machine learning algorithms to two different domains:

I would recommend you use the Python code and associated machine learning algorithms in this tutorial as a starting point for your own projects.

Finally, keep in mind our five-step process of approaching a machine learning problem with Python (you may even want to print out these steps and keep them next to you):

By using the code in today’s post you will be able to get your start in machine learning with Python — enjoy it and if you want to continue your machine learning journey, be sure to check out the PyImageSearch Gurus course, as well as my book, Deep Learning for Computer Vision with Python, where I cover machine learning, deep learning, and computer vision in detail.

To download the source code this post, and be notified when future tutorials are published here on PyImageSearch, just enter your email address in the form below.

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!